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C  A  U  ^  O  I  N  I  A 


PRn^J.MS  IN  THE  THECBY  CF  D^IIAMIC  PROGRAMMIMG 
Richard  Dell man 


fluMuan.  The  theoary  of  dynaMc  progri'iuzulng  treats 
probiema  involving  oultl-stage  processes  by  naans 
of  a  transfomatlon  of  the  problem  from  the  space 
of  decisions  to  the  space  of  functions.  This  is 
accoopllshad  by  deriving  a  functional  equation 
whose  solution  is  equivalent  lo  the  solution  of 
the  original  problem.  To  illustrate  this  approach 
Bost  clearly,  free  of  extraneous  analytic  details, 
we  uuusittei  a  simple  but  nontrivial  multi-stage 
investsient  probleffi;i  We  show  how  exact  solutions  may 
be  obtained  in  soo4  cases,  approximate  solutions 
in  others,  and  how  these  epproxlBate  solutions  may 
be  used  to  obtain  more  accurate  solutions  in  the 
general  cas^  Of  particular  in^ortance  is  the 
decrease  in  the  nuaber  of  independent  variables 
Bade  possible  by  this  approach.  Ihls  is  not  only 
Ijq^artant  froa  the  theoretical  standpoint,  but  is 
also  of  great  value  in  reducing  the  cost  in  time  and 
effort  of  numerical  cojif>utation. 


1.  Introduction. 

The  puiT.>ose  oi  this  paper  is  to  provide  an  introduction  to  a  class  of 
aathenatical  techniques  that  are  useful  in  treating  a  variety  of  problems 
arising  in  the  planning  of  multi-stage  processes.  These  are  programming 
problems,  to  use  the  currently  j-opuiar  terminology,  and  the  adjective 
"dynamic"  emrjhaslzes  tloit  ii.  these  probioms  time  plays  an  in^jortant  role. 

This  characterization  of  OoT  subject  xaatter  is  not  merely  one  of  nomenclcture- 
the  problems  before  us  are  of  a  conceptually  distinct  type.  Moreover,  the 
mathematical  techniques  eaq^loyed  prove  to  be  especially  powerful  for  the 
resolution  of  problems  of  this  type. 

The  multi-stage  problems  in  which  we  are  interested  are  coii|3osed  of 
sequences  of  operations  in  which  the  outcome  of  preceding  operations  may  be 


i. 
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used  to  guide  the  course  of  future  operations.  Ivo  types  of  operations 
nfti'  be  distinguished  inned lately,  those  in  which  the  outcons  is  predict¬ 
able  on  the  basis  of  a  proliabillty  distribution,  and  those  in  which  the  out¬ 
come  is  completely  determined.  Depending  upon  one's  point  of  view,  either 
type  may  be  considered  to  be  an  approximation  to  the  reality  represented  by 
the  other. 

Since  most  of  the  j^roblemc  which  arise  are  of  an  entirely  novel  char¬ 
acter,  frequently  oi'foriiig  forcdduble  uatlieiuatical  difficulties,  we  have 
restricted  ourselves  to  the  consideration  of  a  siii?)ie  yet  important  problem 
in  order  not  to  obscure  our  techaiques  hy  extraneous  analytic  and  algebraic 
eoiq:lications. 

ihe  basic  idea  underlying  our  analysis  is  that  of  replacing  tJje  decision 
problem  by  a  functional  equation.  We  show  then  that  the  solution  of  the 
functional  equation  yields  a  solution  of  the  criglnal  prograimning  problem. 

It  is  precisely  at  this  point  tliat  the  question  ol’  uniqueness  of  solution 
acquirer  in  economic  as  welx  as  a  nathenatical  importance. 

Uiin^;  ttie  fimctionai  equation  approach  we  consider  a  very  siaq^le  version 
of  an  oj  tinal  aiiocaticn  rrcblom,  We  begin  with  a  statement  of  the  problem 
and  then  contrast  the  classical  approach  with  tlxj  dynamic  programming  approach. 
We  next  supply  an  existence  and  uniqueness  proof,  and  then  derive  Bone  iii^or- 
tant  firoperties  of  the  solution, 

Follcwin^q  this  w  formulate  the  stochastic  version  of  the  same  problem, 
and  then  in  the  concludiTig  section  discuss  the  essential  features  of  the 
mathematical  ueenniques  eo^ioyed. 
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2.  Th>  FonoiLatloD  of  a  Problea  of  Optimal  Resource  Allocation. 

A  very  ciin;)le  proMea  relating  to  the  allocation  of  resources  is  the 
following:  We^  have  x  doilarc  which  i»y  be  split  into  two  parts,  y  and  x^y. 
From  y  we  obtain  a  return  of  g(y) ,  froa  »-y  a  return  of  h(x-y) .  How  does 
one  choose  y  so  as  to  oaximlze  the  total  return? 

The  analytic  problem  is  that  of  finding  the  ooximum  of  g(y)  h(*-y) 
subject  to  the  constraint  0  <  y  ^  x. 

Let  us  now  conplicato  the  probiem  by  changing  this  on»*stage  problea 
into  an  K-stage  problem  in  the  following  way;  At  the  end  of  first  stage 
let  us  assume  that  we  have  loft,  as  a  result  of  o’jt  division  into  y  and  x-y, 
the  quantity  of  money  ay  ^  b(x-y),  where  C  <  a,b  <1,  and  we  are  to  continue 
this  process  for  [1-1  additional  stages.  How  does  one  allocate  at  each  stage 
in  order  to  maximize  the  total  return? 

There  is  no  difficulty  in  settijag  tnis  problem  up  in  classical  form, 

^2*  ***»  ^  sequence  of  choices  at  each  successive  stage.  The 

total  return  will  be 


N  H 

(2.1)  ^  ^  h(x.-y.) 

where 

(2.2)  =  X,  C  <  yj^  <  x^, 

X2  =  Vi  b(x^-y^) ,  b  <  y.  £  x^ 


C  4  y,;  i  XK 
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The  problem  la  to  maximize  R  subject  to  the  above  constraints.  Since 
several  of  the  y  may  be  0  or  endpoints  of  their  allowable  intervals,  any 
naive  application  of  calculuo  is  a  bit  hazardous. 

3.  The  Functional  Equation  Approach  to  Optimal  Allocation. 

Since  the  above  formulation  presents  all  the  ux^leasant  features  usually 
involved  in  a  maximization  problem  over  an  N-dimenslonal  region,  let  us  cast 
about  for  an  alternative  approach.  The  clue  to  another  fonoulation  is  the 
observation  that  at  any  stage  it  is  necessary  only  to  choose  the  carrespond- 
Ing  y  in  order  to  continue  the  process. 

Let  us  set 

(3.1)  "  total  return  obtained  from  an  K-stage  process  starting 

with  X  dollars  and  using  an  optimal  procedure. 

Whatever  division  of  x  into  y  and  x-y  is  optimal  on  the  first  step,  the  remaii*- 
ing  amount  of  money  ay  -f-  b{x-y)  must  be  treated  by  an  optimal  procedure  for 
the  next  IJ-1  steps  if  a  maxinnun  return  is  to  be  obtained  for  N  steps.  Thos 
the  return  for  the  H-step  process  due  to  this  initial  division  is  g(y)  -f- 
b(x-y)  -p  fj^j_(ay  •♦-b(x-y) ) ,  Since  an  optimal  selection  of  y  subject  to 
0  ^  y  ^  X  would  maximize  this,  we  must  have  the  equations 

(3.h)  f^(x)  -  Max  []g(y)  t-  h(x-y)^  , 

Oli’SX 

fj.(x)  =  Max  rg(y)  h(x-y)  t  ^(x-y))!  (N  >  2). 

tiyix 

We  have  thus  re^iuced  tiie  original  programming  problem  to  the  solution 
of  tho  system  of  functional  equations  given  in  (3.f). 

If  we  are  interested  in  the  case  vliere  I.  is  large,  we  obtain  a  great 
sia^  lif ication  by  making  tho  approximation,  f;;(x)  ^ 
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unbounded  number  of  0])eratlone,  vrhich  Is,  of  course,  neaninc^so  in  any 
practical  situation,  possesses  a  very  Ic^ortant  nathenetical  property  of 
invariance,  because  after  any  finite  niunber  of  initial  operations,  there 
still  remains  a  process  with  an  infinite  number  of  steps. 

Introducing 

(3.3)  f(x)  a  total  return  obtained  from  tho  infinite  process  starting 

with  a  quantity  x  aid  using  an  optimal  policy, 

we  obtain,  formally,  the  equation 

(3.4)  f(x)  =  Max  [g(y)  ^  h(x-y)  +  f(ay  +  b(x-y))"] 
for  X  i  0. 

ihe  advantage  of  the  above  equation  over  the  system  in  (3.1)  lies  in 
the  fact  that  we  have  now  a  single  unknovm  function.  Ihe  disadveuitage  lies 
in  the  difficulty  usually  associated  with  infinite  processes,  namely  that  of 
proving  existence,  uniqueness,  and  attninal  ility.  l-fe  shall  show  in  the  next- 
section  that  in  this  problem  this  is  a  minor  difficulty. 

4.  nxistonce  and  Jnirueness. 

Here  we  establish  t-he  existence  and  uniqueness  of  the  solution  of  (3.4), 
imder  certain  natural  assun^-tlons  of  continuity.  'Ihe  method  of  successive 
approximations  wnich  we  emrloy  is  equlvedent  to  ^-he  obvious  idea  of  showing 
that,  as  N  — >  oo  we  have  fj'(x)  f(x).  there  ore  alternative  approximation 
techniques,  also  based  aj,on  economic  principles,  which  we  discuss  elsewhere. 


continuoufi  throughout  the  whole  Inf  rv4l  • 

As  v/e  shall  see  from  the  proof  the  coDdltion  that  g  aad  h  are  monotone 
increasing  is  not  needed.  Use  of  this  condition,  howver,  wldch  is  satisfied 
in  any  application,  sia^lifies  the  proof  notationally. 


jroof ;  We  use  the  successive  approxinaticiui  defined  by  (3.2).  Since 
f 0  it  is  clear  from  (3.2)  that  f^  i  f^,  which  ylexds,  via  an  induction, 
the  obvious  result  that  C  ^  f ^  S.  f2  <  *•*  <  Let  c  ■  Max(a,b)  and  let 

us  show  that 
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iience  the  sequence  is  oonotone  increasing  and  bounded,  which 
means  that  P^(x)  convcrces  to  a  function  f(x).  Ihe  function  satisfies  the 
equation 


(4.4)  f(x)  »  Sup  u:(y)  -*•  f  f(ay  4-  b(x-y))']  . 

In  order  to  prove  that  the  maximum  is  attained,  we  must  pra/e  that  f 
is  continuous,  lo  accon^iish  this,  we  use  the  fact  thr:t  each  f^  is  a  con¬ 
tinuous  function  of  x,  and  establish  the  unilorm  conver^^ence  of  the  series 
-  fyj|  by  means  of  the  following  technique: 

Let  ^  *  point  where  the  maximum  is  attained  in  the  expres¬ 
sion  for  i*5*» 


=  c(yn)  t  h(x-yj^) 


Then  we  iiuve  tne  following  obvious  inequalities. 


(4.0 


C(yn^  h(x-yjj) 


t-Ux-jp) 


>  o(yn)  +  i-(x-yp  t  l'^i(»yn  +  l^ix-y^)). 


It  follows  thiat 
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^iV4l  -  i  +  »(x-y^^))  -  +  l>(x^y^i)) 

i  V*J'n  + )  -  ^n.i<*»'n  1-b(*-yn)). 


Hence 

(4.S)  r^^(x)  -  yx)  <  f^[|  V«Vl  ' 

Let  for  nil 

(4.9)  u  (x)  s  i-lax  |f.+.i(y)  -  • 

C^^fpc  '  ^ 

From  (4.8)  we  obtain 

(4. 1C)  ‘  ^(cx). 

Hence  ^jj^(x)  £  u^(cx)  for  n  >1,  while  for  n  =  0,  ve  have 

(4.11)  u  (x)  -  Max  f,(y)^-  l(x)  •4-h(x). 

o<^.«  ■" 

From  Lhi3  we  conclude  that  the  ceriee  ^  j.  is  majorized  by  the  series 

n 

^  |c(c^)-h  h(c*V)J  .  Hence  the  series  converges  uniformly  in  the 

inteirval  jCjxJ,  wldch  means  that  f^  convercos  uniformly  to  f (x) ,  v^icb  mist 
then  be  continuous. 

This  coi^^lotos  the  ;  roof  of  existence.  To  establish  uniqueness,  we 
I  roceed  similarly.  Let  F(x)  be  anotlier  solution  of  the  equation  which  is 
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coni^inuouc  at  x  =  0.  Iher  as  in  (4.S)  -  (4.*  )  vre  obtain  the  inequality 

(4.12)  |f(x)  -  F(x)|  ^  I-lax  [^f{ay-l'  L(x-y))  -  i  (ay  -t-tlx-y))  , 

f{az  l(x-z))  -  F(az  b(x-z))^. 

where  y  =  y(x)  is  a  point  at  which  the  maximini  of  ^^(y)  ‘‘(x-y)  4-  f  (ay  b(x-y) ) 

ic  attained  euid  z  is  a  corresponding'  point  acsociatod  tllh  F.  Defining; 
v(x)  =  Sup  jf(y)  -  F(y)l  we  ob  ain  froja  (4.J-4)  the  series  of  inequalities 

(4.13)  v(x)  4  v(cx)  L  v(c‘^)  <^v(c^)  (npi,;. ,*  ••)• 

Since  v(x)  is  continuous  at  C  and  has  the  value  0  there,  we  see  that  as 
n  — ^  oo  (4.13)  leads  to  the  conclusion  that  v(x)  is  idonticuily  zero  through- 
or,  [o,*„]  . 

ihis  coni^letes  tlie  piToof  of  existence  and  uniqueness. 

t .  yumly'tic  Resuxls  ;;  and  h  both  Convex. 

Since  tlje  basic  functional  equation  is  nonliiKjar,  it  is  too  laach  to 
e^qiect  that  the  solution  may  in  general  be  obtained  ii.  exf-licit  form.  Instead 
wc  must  focus  our  attention  upon  derivir^:  siiu}  ±e  general  qualitative  proper¬ 
ties  of  the  solution  t'rom  sinif  le  ascum:  Lions  concerning  g  and  h.  The  term 
solution  is  used  to  incl'ude  bov-n  the  func'ion  satisfying  the  1'unc‘ional  equa¬ 
tion  and  the  ;  oiicy  yielding  the  optinal  retuin.  .he  fact  that  this  duality 
exists  ic  of  tremendous  Imjjortance  ii;  connection  with  bo„h  theoretical  and 
coiq^utational  invostigations. 

In  this  section  we  ob^^ain  a  cinif^ic  consequence  of  convexity. 


P-455 

-10- 


'I  heorea  If 


(5.1)  (a)  ti(0)  =  h(0)  -  G, 


(b)  c*(x),  ;!»{x)  >C,  g"(x),  n"(>:)  >  C  for  0  -C  x  £x  ,  ^ 


1 c^*x)  -*■  h(c*^)^  ^ 


where  c  =  Max  (a,b) ,  t.he  optimal  i)olicy  consists  in  choosing  y  *  C  or  x 


for  X  in 


iroof ;  'ihe  i.roof  is  readily  obtained  by  showing  that  each  of  the  functions 
f  is  convex.  It  is  clear  that  g(y)  -h  h(x-y)  is  convex  an  a  function  of  y 

ti 

for  0  <  y  <.  X.  Hence 


(i.2) 


-  Max  |T;(y)  *■  h(x-y)]  =  Max  [i:(x),  h(x)^  . 

C  oc 


From  this  it  follows  tJia*.  r^(x)  is  convex,  anti  hence  tliat  g(y)  h(:>*y) 
+  ^(^y))  *  function  of  y  is  convex  in  Qc,xJ  .  Thus 


(C.3)  f^,(x)  a  Max  Q  (x)  h(x)  -Hf^(bx)J  , 


which  is  a(’fiin  a  convex  function.  J.  is  clear  r.ov  that  the  general -re suit 
follows  inductively. 

tlie  resulting  functional  eejuation 


(:j.4)  f  ^x)  -  Max  Q:^x)  4-l’(ax),  h(x)  ''(lx)J 


be  scxved  in  ccr  aii.  .pnecial  c  whici:  wr  shalu  not  enter  in  o  liere, 
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6.  Analytic  Resulta. 

Let  us  nov  return  to  the  equation 

(C.l)  f{x)  a  Max  r  g(y)-+-  h(xr-y) f  (ay b(x-y)^  f(0)  *  0 

and  aasume  that  g  and  b  are  both  concave  Inareaeing  functions  of  x.  The 
problen  is  nov  much  siore  cooplex,  and  In  general,  the  optlnel  y  will  not  be 
at  an  end  point. 

We  shall  prove 

Theorem  3.  l£t 

(a)  c(C)  =  h(C)  a  C, 

(6.2)  (b)  g*  (x) ,  h*(x)  >  0  for  x  iO, 

(c)  c”(x),  h"(x}  >  0  for  X  >^0, 

and  consider  the  sequence  of  to  f  defined  b,Y 

f  (x)  =  Max  Tgly)  4-  h(x-y)~| 

°  C<^<x  ^ 

(6.3) 

f^  j^(x)  =  Max  r  L(y) 6(x-y)  f  (eg’ -fr*  b(x-y)  )1 , 

C  6- 

For  each  n,  there  is  a  unique  y  =  y  (x)  that  yields  tlie  naxinBim.  6  ^^a, 
vB^Jia^  ^  <  y^‘  •  •  t  and  the  reverse  inequalities  for  b  ^  a.  In  parti¬ 


cular.  if  yj-^(x)  =  X  for  30CE  n  in  the  case  i-  S  a,  liten  yjj(x)  =  x  £21  ®  ^ 
and  the  solution  of  the  equation  in  {<  .1)  will  be  furnished  by  y  =  x 
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This  r«8\ilt  is  iaporXMxit  in  connsction  with  dstsraining  appraxiasts 
solutions,  sinca  it  is  quits  slj^le  to  detondns  y^,  ^2*  ^3 

cally. 

Us  shall  bs^^iXi  by  aosiunin^  all  the  raaxiiaa  occur  vitiiin  the  inteiv 
val  |o,:^  and  shall  then  consider  the  case  in  which  one  y^^ix)  ■  x.  Consider¬ 
ing  the  function  f^^tx),  we  see  that  its  aaxiimun,  y,  is  deteroined  by  the 
equation 

(<-.4)  g'(y)  =  h*(x-y). 

Since  the  left-hand  side  is  aonotone  increasing  and  the  right-hand  eida  is 
monotone  decreasing,  there  Is  at  most  one  solution.  If  we  assume  h* (x)  > 
g'(C),  g’(ac)  >  h'(0),  there  will  be  exactly  one  solution  of  (6.4),  which  we 
call  y-i^  «  y^^tx).  Differentiating  (6.4),  we  obtain 

(^.5)  y2^g"(yi)  =  (1  -  yi)h"(x  -  y^), 

which  yields 

h"(x  -  y^) _  ^ 

g"  (y^) h"  (x  -  y^^) 

and 

(6.7)  1  -  y*  >  0. 


Turning  to  the  expression  for  f,  ws  have 
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(6.8)  f^(x)  =  g(y^)  4-h(x  -  y^) , 

whtnce 

(6.9)  f|(x)  =  g‘  (y^)  +•  (1  -  yj)h*  (x  -  y^)  =  h»  (x  -  y^) , 

using  (6.4).  Thus  f|(x)  >  0  and  f^(x)  -  (1  -  y|)h"(x  -  yj^)  4  0,  which  dmuis 
that  fj^(x)  la  concave. 

Let  UB  now  turn  to  the  function  ^2  (^c)  > 

(6.10)  fp(x)  *  Max  g(y)  h(x-y) -t- f,  (ay -h  b(x^y))  . 

/.s Burning  that  there  la  a  maximum  Inaide  the  interval,  we  obtain 

(6.11)  e'(y)  -  h»(x-y)  (a-b)f|(ay-h  b(x~y))  -  C, 
which  W0  write 

(6.12)  g'(y)  -f  (a-b)f|(uy  b(»-y) )  =  h»  (x-y) . 

"ihe  left-hand  aide  is  again  strictly  decreasing  and  the  right-hand  aide 
strictly  Increasing,  so  that  there  is  at  most  one  solution  which  we  call 
yr  ■  yol^c)*  IP  it  exists.  IJote  that  if  there  is  no  solution  of  (6.12),  Uien 

(o.l3)  f2(x)  ■  c(x)  -f-  f^(ax). 


Let  UB,  however,  assume  that  there  is  aaolution.  Then 
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(6.U)  *  6(y2)  -p  -  y2)  f^(ay2-*'^(x  -  y^))! 


wbtnce,  as  above,  using  (6.12), 


(C.15)  f^^x)  =  h»(x  -  y^)  ^l(®y2  “  ^2^^ 


Using  (6,12)  again,  this  my  be  written 


(C.16) 


ft  ,  ah«(x  «  72)  -  bg*(y2) 


a  -  b 


This  procedure  la  perfectly  general,  and  we  obtain,  under  our  aosuaptlon 
concerning  the  existence  of  an  Internal  loaxloua. 


ah*(x  -  y  )  -  he'(yJ 

(0.17)  f*  = 
n 


a  -  b 


n-1,2,3,*'-. 


We  now  wish  to  show  that  If  b  4.  a,  then  y^c  y^S****  oonvereely. 
If  a  <  b,  that  yj^  >  y2  ^  *  *  *  •  cases  are  really  one,  since  we  nay 
Interchange  the  roles  of  y  and  x  -  y  if  we  so  wish.  Since  >  0,  we  see, 
on  cojujarlng  (6.12)  and  (6.4),  that  y_j^<  y2« 

The  equation  for  y^  1g 


(C.18)  r*(y)  +  (e-b)f^(ay  +■  b(x-y))=  h'(»-y). 


If  we  can  show  that  f2(x)  >  f|(x),  the  sane  argunent  as  that  for  yj|^»y2 
shows  that  y^  >  y^.  Con?>arlng  (6.8)  and  (6.15),  we  see  that  f^  >  f|,  since 
h'(x-y2)  >  ^'(x-yj^). 

To  obtain  the  result  for  general  n,  always  assuoing  that  the  occur 

at  inner  poijits,  we  use  (6.17).  We  know  that  fn(x)  >  that 

yn4i  >  yn*  Since  the  function 
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fth‘(x-y)  -  bg‘(y) 

(6.19)  r(y)  =  -  ' 

a  -  b 

Is  monotone  increaslnc  In  y  and  y^  >  inductive  hypothesis,  it 

follows  that  y  ^xH-l"^  ^n* 

Let  us  now  consider  the  situation  in  which  some  y  (x)  ■  x.  If  n  =  1,  it 
is  easy  to  see  that  y^{x)  «  x,  n  ^  1,  since  yj^(x)  •  x  means  tljat  c'(y)  2.  -y) 

for  0  <  y  <  X,  Since 

(^.20)  jg(y)  -H  h(x-y) -»*  f^(uy -h  b(x-y)  j] 

=  g*(y)  -  h'(x-y)-f  (a^b)f*(ay b(x-y)) 

and  a  >  b,  we  see  that  this  expression  is  positive  if  g'(y)  ^h’(x-y)  for 
0  4:  y  Hence,  ^2^^^  *  similarly,  y^(x)  •  x. 

Let  us  now  tsike  the  case  in  wnich  y2(x)  -  x,  yj^(x)  ^  x.  Since  y^C*)  -  x 
is^lies  that  g*  (y)  -  h’  (x-y)  (a-b)f^(ay  b(xr«y))  >  0  for  all  0£^y  ^x, 
we  have,  in  particular,  g'(x)  -  h'(C)  -f  (a-l)f^(ax)  2 

(6,21)  f2(x)  ■  g'(x)-»'  af*(ax) 

=  g'  (x)  -  h’  (0)  (a-b)f|(aj.)  h'  (C)  -f-  bf  j(ax) 

2  h'(G) 

and  f!(x)  =  h'(xF-yv)  C  h'(0),  we  see  that  f'(x)  ^  f ’  (x) .  This,  os  above,  implies 

J  •  ^  A 

that  yj  >y2  ~  process  continues, 

let  us  note,  finally,  that  if  c*(y)  >  h' (x-y)  for  all  y  in  [o,xJ,  then 
e'(y)  2  h’(2-y)  for  y  in  [o,i^  for  all  z  ^x. 

In  closing,  this  discussion  of  the  functional  equation,  let  us  observe 


that  if  an  interior  naximui.  exists,  wo  must  have 


((^  .2.':)  g*(y)  -  ii'(x-y)  -j-  (»-b)f * )  =  c. 


and 

(<-  .23)  f '  (x)  =  h'  (x-y)  4"  bf '  (ay  +  b(»-y)) . 

''.his  systen  of  functional  equations  for  y  and  f(x)  may  be  solved  explicitly 
if  ^  and  h  are  quadratic,  which  is  a  fact  of  some  '-.se  in  ootainin^  approxi¬ 
mate  solutions.  In  general,  however,  tlie  system  does  not  seem  to  be  of  much 
use  for  this  particular  equation,  iiowever,  for  other  closely  related  equa¬ 
tions,  equations  similar  to  (c.22)  and  (C.23)  play  a  very  iji?)ortant  role  in 
ietermining  the  solution,  as  we  shall  show  elsewhere. 

7 .  Stochastic  Case . 

Thus  far  we  iiuvo  considered  a  situation  in  wnich  the  outcome  of  a  por- 
Mcular  division  of  resoorcos  is  completely  determinate,  uet  os  now  briefly 
sketch  the  modifications  required  to  treat  the  case  where  there  is  a  proba¬ 
bility  distribution  oi'  outcomes,  lat  us  assume  if  a  division  into  y  and  x-y 
is  made,  there  ij  a  prohauility  p^,  which  in  some  cases  might  very  well  be  a 
function  of  y,  that  tJio  return  will  be  t^(y)  -f'n^(x-y)  with  a^y  -l"b^(x-y) 
dollars  avaiialic  lor  distribution  iii  the  next  stage,  and  a  probability  p,^ 
that  the  return  will  lie  g  ,(y)  4-  h_^(x-y)  with  b  (x-y)  dollars  remaining. 

Jince  we  are  dealing  with  stocrjastic  variables,  it  is  necessary  to  intro¬ 
duce  the  metric  of  probability  theory  and  speak  of  expected  values.  Let  us 
define 

(V.i)  f(x)  -  exj'ecied  total  return  stATtin^-  with  x  dollars  and  using 

an  oj  t.iaai  pox  icy  for  an  iif  inite  number  of  stages. 
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I'sinc  thu  Bane  ar^; aments  as  lefore,  we  see  tliat  (formally  at  least) 
f(x)  eatisfios  the  functional  aquation, 

.  )  f(x)  =  Max 
C^4X 

P2  i’(a2>y  b^(x-y)^ 

Precisely  the  same  teciuilques  as  ap}  lied  above  may  now  be  used  to  show  exist¬ 
ence  and  uniqueness,  aid  to  derive  results  coi:»*espondinc  to  the  previous 
anal^’tlc  results. 

8.  Conclusion. 

i-et  js  now  atten;.i  to  abstract  some  of  the  essential  features  of  the 
preceding  problem,  features  which  are  common  to  a  larpe  class  of  problems 
amenable  to  the  teclmiques  of  the  theory  of  dynamic  pro^pramminc . 


I  ^^1  b_j^(xr-y)  -b  f(aj^y  -bb^Cx-y)^ 


(6.1)  (a)  We  have  an  economic  system  described  at  any  time  by  a  set  of 

j:>aruEietor.i ,  ;,  wi.ich  wc  call  the  state  variables. 

(b)  ,\t  certain  ’Lacs  \>k  are  t,o  choose  one  of  a  set  of  decisions, 

L,  wiiich  will  nave  tlio  eflect  of  tranaformin^^  these  state 
varlaLio--  into  u  sicij.ar  set.  The  outcome  of  a  decision  may 
or  may  no'  be  comj  letely  known. 

(c)  The  choice  of  decisions  Is  (governed  by  the  desire  to  maximize 
some  function  of  the  fir<al  state  variables,  a  criterion  functioix. 


What  is  desircw  is  a  rule  whici*  wixl  yicxu  the  oj  timal  decision  at  each 
staje.  Knowing:  the  values  of  'he  state  variatles  ami  t>ie  permissible  decisions 
at  that  sta^^e. 

In  the  problem  treated  above,  there  uere  for  tne  IT-stapc  process  three 
state  variables,  naacj-j  (a)  tlie  juantlly  of  money  available  for  division  at 
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oach  sta^-e,  (b)  the  number  of  remaining  stages,  and  (c)  the  retum  obtained 
from  the  preceding  stages.  Our  aim  was  to  maximize  the  state  variable  of 
(c)  at  ihe  end  of  the  process. 

We  have  purposely  left  the  description  a  bit  loose,  since  we  feel  that 
it  is  the  spirit  of  the  technique  that  is  iiH)ortant.  Once  grasped,  the  prem¬ 
ier  formulation  of  any  particular  problem  becomes  a  matter  of  ingenuity,  a 
•uality  which  cannot  be  altogether  banished  from  oetheaatics. 

In  order  to  obtain  a  mathematical  formulation  of  the  general  problem, 
let  us  define 

(8.2)  f(P)  =  the  function  of  the  final  state  variables  obtained 

using  an  optimal  policy  starting  with  the  initial 
variables  represented  by  P, 

lyCt  us  also  represent  the  transformation  effected  by  a  choice  by  P*  ■ 
where  k  represents  the  parameters  describing  the  particular  choice.  To 
ol  tain  a  functional  eu/uation  governing  the  process,  we  use  the  following 

obvious 

Principle  of  Optimalit:/;  /ai  optimal  policy  has  the  property  that  whatever 
the  initial  state  ard  the  initial  decision  may  be,  the  remaining  decisions 
niisl  constitute  an  optimal  ]Joiicy  with  regard  to  the  state  resulting  from 
the  first  decision. 

The  matJiematiwil  translation  of  this  statement  is  the  equation 

(P.3)  f(P)  -  Max  f(';j^(P))  . 

k 

Ibis  eq'^tion  may  now  be  used,  as  in  tl*  preceding  sections,  to  determine 
various  proparties  of  optimal  policies  .uder  suitable  assuiq  tions  concerning 
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T  (P) .  Of  particular  ii^oortazico  from  the  mthaimitlcal  and  practical  view- 
points  are  those  criterion  functions  that  possess  Invariant  properties  and 
that  reduce  the  number  of  state  variables  required.  Hjq)ected  return  Is  an 
ii^oirtant  function  of  this  type,  perhaps  the  most  ij^ortant,  since  a  know¬ 
ledge  of  the  return  over  the  preceding  stages  is  not  needed  in  order  to 
determine  the  optimal  continuation.  Infinite  processes  are  Irportant  for 
a  like  reason,  since  the  number  of  stages  remaining  stays  constant. 

A  more  extensive  and  Intensive  study  of  the  various  types  of  functional 
equations  arising  from  (8.3)  will  appear  subsequently,  [jt],  [  vj  .  The 
interested  reader  may  also  wish  to  consult  the  references  listed  below. 
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